Various types of software have been developed to execute on the many different types of computer systems that are available. These different types of software range from operating systems (OS), firmware, and other low level applications, to application programs. Moreover, certain software can be classified as a software platform, in that other software can execute on this platform software. While one example of a software platform may be an OS, other examples can be any type of platform such as a cloud computing platform on which application software executes.
A software application that depends, for its proper operation, on the services of an underlying platform, can be said to “target” that platform. A given platform, and all of the applications that target it, can be called a “platform ecosystem.” The services of the platform are exposed to the ecosystem's applications through an Application Programming Interface, abbreviated API (which may take many forms, such as a software library, a SOAP interface, an RESTful interface, etc.). An application that targets a given platform can be called a “Platform-Targeting Application”, abbreviated PTA. The independent entities that develop PTAs can be called the platform's “Independent Software Vendors,” abbreviated ISVs.
By way of example, the Netflix application (of which Netflix Inc is the ISV) is a well-known PTA. Versions of the Netflix client target a variety of platforms, while the Netflix application's server-side implementation targets the Amazon Web Services cloud computing platform.
Under the methodology known as “continuous integration,” whenever a software product's implementation is changed, the changed product is subjected to a suite of rigorous automated tests defined by the product's developer. One kind of testing, called “regression testing,” tests the changed software product to see if the new changes break existing functionality. For applications, such testing by the product's developer, called “internal” testing, is sufficient to find product bugs. However, changed platforms require additional testing—of both its implementation and of its API—to ensure that the change does not break applications that target the ecosystem. If a bug in a platform keeps the platform itself from working, then it is a product bug. If, however, it does not keep the platform from working, but breaks one or more of its PTAs, then it can be called an “ecosystem bug.”
The traditional approach to finding ecosystem bugs is “beta testing.” In beta testing, once a new version of a platform—incorporating many changes—has passed all (or most) of its own test suite, it is made available to the platform's ISVs. ISVs can use the beta version of the changed platform to test their PTAs, to see if any of its many changes break their PTAs.
There are three main problems with the beta testing process, as historically implemented in the computer software industry. First, it waits to test many changes at once, at the very end of the development process. This is inefficient, because if a bug is found when it is first written, it tends to be easy to fix, whereas if undetected, additional code may be layered over the unfound bug, making it much more difficult to find and fix later. Second, it tests a multitude of changes simultaneously, which means that many different bugs may interact in complex, hard-to-identify, and even harder-to-fix ways. Third, it places a large source of schedule risk at the very end of the development process, when the greatest amount of schedule certainty is needed (to coordinate sales and marketing campaigns around the new version of the platform).